Self-supervised learning for transferable representations

Machine learning has undeniably achieved remarkable advances thanks to large labelled datasets and supervised learning. However, this progress is constrained by the labour-intensive annotation process. It is not feasible to generate extensive labelled datasets for every problem we aim to address. Consequently, there has been a notable shift in recent times toward approaches that solely leverage raw data. Among these, self-supervised learning has emerged as a particularly powerful approach, offering scalability to massive datasets and showcasing considerable potential for effective knowledge transfer. This thesis investigates self-supervised representation learning with a strong focus on computer vision applications. We provide a comprehensive survey of self-supervised methods across various modalities, introducing a taxonomy that categorises them into four distinct families while also highlighting practical considerations for real-world implementation. Our focus thenceforth is on the computer vision modality, where we perform a comprehensive benchmark evaluation of state-of-the-art self supervised models against many diverse downstream transfer tasks. Our findings reveal that self-supervised models often outperform supervised learning across a spectrum of tasks, albeit with correlations weakening as tasks transition beyond classification, particularly for datasets with distribution shifts. Digging deeper, we investigate the influence of data augmentation on the transferability of contrastive learners, uncovering a trade-off between spatial and appearance-based invariances that generalise to real-world transformations. This begins to explain the differing empirical performances achieved by self-supervised learners on different downstream tasks, and it showcases the advantages of specialised representations produced with tailored augmentation. Finally, we introduce a novel self-supervised pre-training algorithm for object detection, aligning pre-training with downstream architecture and objectives, leading to reduced localisation errors and improved label efficiency. In conclusion, this thesis contributes a comprehensive understanding of self-supervised representation learning and its role in enabling effective transfer across computer vision tasks

How Well Do Self-Supervised Models Transfer?

Self-supervised visual representation learning has seen huge progress recently, but no large scale evaluation has compared the many models now available. We evaluate the transfer performance of 13 top self-supervised models on 40 downstream tasks, including many-shot and few-shot recognition, object detection, and dense prediction. We compare their performance to a supervised baseline and show that on most tasks the best self-supervised models outperform supervision, confirming the recently observed trend in the literature. We find ImageNet Top-1 accuracy to be highly correlated with transfer to many-shot recognition, but increasingly less so for few-shot, object detection and dense prediction. No single self-supervised method dominates overall, suggesting that universal pre-training is still unsolved. Our analysis of features suggests that top self-supervised learners fail to preserve colour information as well as supervised alternatives, but tend to induce better classifier calibration, and less attentive overfitting than supervised learners.Comment: CVPR 2021. Code available at https://github.com/linusericsson/ssl-transfe

Better Practices for Domain Adaptation

Distribution shifts are all too common in real-world applications of machine learning. Domain adaptation (DA) aims to address this by providing various frameworks for adapting models to the deployment data without using labels. However, the domain shift scenario raises a second more subtle challenge: the difficulty of performing hyperparameter optimisation (HPO) for these adaptation algorithms without access to a labelled validation set. The unclear validation protocol for DA has led to bad practices in the literature, such as performing HPO using the target test labels when, in real-world scenarios, they are not available. This has resulted in over-optimism about DA research progress compared to reality. In this paper, we analyse the state of DA when using good evaluation practice, by benchmarking a suite of candidate validation criteria and using them to assess popular adaptation algorithms. We show that there are challenges across all three branches of domain adaptation methodology including Unsupervised Domain Adaptation (UDA), Source-Free Domain Adaptation (SFDA), and Test Time Adaptation (TTA). While the results show that realistically achievable performance is often worse than expected, they also show that using proper validation splits is beneficial,as well as showing that some previously unexplored validation metrics provide the best options to date. Altogether, our improved practices covering data, training, validation and hyperparameter optimisation form a new rigorous pipeline to improve benchmarking, and hence research progress, within this important field going forward

Why Do Self-Supervised Models Transfer? Investigating the Impact of Invariance on Downstream Tasks

Self-supervised learning is a powerful paradigm for representation learning on unlabelled images. A wealth of effective new methods based on instance matching rely on data augmentation to drive learning, and these have reached a rough agreement on an augmentation scheme that optimises popular recognition benchmarks. However, there is strong reason to suspect that different tasks in computer vision require features to encode different (in)variances, and therefore likely require different augmentation strategies. In this paper, we measure the invariances learned by contrastive methods and confirm that they do learn invariance to the augmentations used and further show that this invariance largely transfers to related real-world changes in pose and lighting. We show that learned invariances strongly affect downstream task performance and confirm that different downstream tasks benefit from polar opposite (in)variances, leading to performance loss when the standard augmentation strategy is used. Finally, we demonstrate that a simple fusion of representations with complementary invariances ensures wide transferability to all the diverse downstream tasks considered.Comment: Code available at https://github.com/linusericsson/ssl-invariance

Self-Supervised Disentanglement by Leveraging Structure in Data Augmentations

Self-supervised representation learning often uses data augmentations to induce some invariance to "style" attributes of the data. However, with downstream tasks generally unknown at training time, it is difficult to deduce a priori which attributes of the data are indeed "style" and can be safely discarded. To address this, we introduce a more principled approach that seeks to disentangle style features rather than discard them. The key idea is to add multiple style embedding spaces where: (i) each is invariant to all-but-one augmentation; and (ii) joint entropy is maximized. We formalize our structured data-augmentation procedure from a causal latent-variable-model perspective, and prove identifiability of both content and (multiple blocks of) style variables. We empirically demonstrate the benefits of our approach on synthetic datasets and then present promising but limited results on ImageNet

Stirring the motivational soup: Within-person latent profiles of motivation in exercise

Background: The purpose of the present study was to use a person-oriented analytical approach to identify latent motivational profiles, based on the different behavioural regulations for exercise, and to examine differences in satisfaction of basic psychological needs (competence, autonomy and relatedness) and exercise behaviour across these motivational profiles. Methods: Two samples, consisting of 1084 and 511 adults respectively, completed exercise-related measures of behavioural regulation and psychological need satisfaction as well as exercise behaviour. Latent profile analyses were used to identify motivational profiles. Results: Six profiles, representing different combinations of regulations for exercise, were found to best represent data in both samples. Some profiles were found in both samples (e.g., low motivation profile, self-determined motivation profile and self-determined with high introjected regulation profile), whereas others were unique to each sample. In line with the Self-Determination Theory, individuals belonging to more self-determined profiles demonstrated higher scores on need satisfaction. Conclusions: The results support the notions of motivation being a multidimensional construct and that people have different, sometimes competing, reasons for engaging in exercise. The benefits of using person-oriented analyses to examine within-person interactions of motivation and different regulations are discussed. © 2017 The Author(s)

Enforcement of Administrative Penalty

Bakalaura darbā “Administratīvā soda izpilde” tiek pētīts administratīva soda izpildes normatīvo aktu regulējums Latvijā un tā problēmas praksē. Darbā tiek pētītas vairākas tiesību normas, kas nosaka administratīvo sodu brīvprātīgo izpildi un administratīvā soda piespiedu izpildi, tiek sniegti arī ierosinājumi kā uzlabot normatīvo aktu regulējumu. Darbā tiek analizēti administratīvā soda izpildes instrumenti un to atbilstība mērķim. Kā arī darbā tiek pētīta administratīvā soda izpildes mijiedarbība ar tiesiskās aizsardzības procesu un maksātnespējas procesu, analizēts iestāžu viedoklis par piemērojamām tiesību normām administratīvā soda izpildes stadijā.In the bachelor's thesis “Enforcement of Administrative Penalty” is studied the regulation of administrative penalty enforcement legislation in Latvia and its problems in practice. In the bachelor's thesis are studied several legal norms that determine the voluntary enforcement of administrative penalties and the compulsory enforcement of administrative penalties and provided suggestions on how to improve the regulation. In the thesis are analysed instruments of enforcement of administrative penalties and their compliance with the purpose. Also, in bachelor’s thesis is studied influence of the enforcement of an administrative penalty with the legal protection process and the insolvency process, analyses the opinion of the state institutions on the application of legal norms at the stage of the enforcement of an administrative penalty

Development of emergency opening for exterior corridors in case of smoke

Arbetets syfte är att undersöka och arbeta fram ett nytt koncept för öppningsmekanismen vid nödöppningssystem av loftgångar. Målet är att ta fram ett koncept som är flexibelt så att detta ryms inom höjden av fönsterluckorna. Detta för att det nya konceptet ej skall vara beroende av loftgångspartiets totalhöjd vilket medför att systemet kan användas inom fler användningsfall. Arbetet är utfört genom att relevanta metoder att arbeta efter valts för att sedan utarbeta de teorier som ansetts vara nödvändiga för att lösa problemet. Därefter har flertalet produktutvecklingsprocesser utförts för att arbeta fram koncept som kan användas för att lösa problemet. Resultatet av arbetet är två koncept som båda uppfyller de kriterier som krävs för en säker och fullständig öppning av fönsterluckorna. Resultatet har sedan analyserats och författarna har jämfört konceptens för- och nackdelar mot varandra samt gett förslag på vidare utveckling

Self-Supervised Representation Learning: Introduction, advances, and challenges

Self-supervised representation learning methods aim to provide powerful deep feature learning without the requirement of large annotated datasets, thus alleviating the annotation bottleneck that is one of the main barriers to practical deployment of deep learning today. These methods have advanced rapidly in recent years, with their efficacy approaching and sometimes surpassing fully supervised pre-training alternatives across a variety of data modalities including image, video, sound, text and graphs. This article introduces this vibrant area including key concepts, the four main families of approach and associated state of the art, and how self-supervised methods are applied to diverse modalities of data. We further discuss practical considerations including workflows, representation transferability, and compute cost. Finally, we survey the major open challenges in the field that provide fertile ground for future work